{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = 'all'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div class=\"jumbotron\">\n",
    "    <h1 class=\"display-1\">GPU与深度学习</h1>\n",
    "    <hr class=\"my-4\">\n",
    "    <p>主讲：李岩</p>\n",
    "    <p>管理学院</p>\n",
    "    <p>liyan@cumtb.edu.cn</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 自2000年以来，GPU性能每十年增长1000倍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 如何使用单个GPU，然后是如何使用多个GPU和多个服务器（具有多个GPU）进行深度学习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 显卡类型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### NVIDIA显卡"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 确保电脑至少安装了一个NVIDIA GPU\n",
    "- 下载并安装NVIDIA驱动 [https://www.nvidia.cn/Download/index.aspx?lang=cn](https://www.nvidia.cn/Download/index.aspx?lang=cn)\n",
    "- 下载并安装CUDA  [https://developer.nvidia.com/cuda-downloads](https://developer.nvidia.com/cuda-downloads)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- `Windows`系统查看`GPU`方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<div class=\"row\">\n",
    "    <div class=\"col col-md-6\">\n",
    "        <center><img src=\"../img/5_deep_learning_computation/winGPUChk1.png\" width=60%></center>\n",
    "    </div>\n",
    "    <div class=\"col col-md-6\">\n",
    "        <center><img src=\"../img/5_deep_learning_computation/winGPUChk2.png\" width=80%></center>\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- `Mac`系统查看`GPU`方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<div class=\"row\">\n",
    "    <div class=\"col col-md-4\">\n",
    "        <center><img src=\"../img/5_deep_learning_computation/macGPUChk1.png\" width=70%></center>\n",
    "    </div>\n",
    "    <div class=\"col col-md-4\">\n",
    "        <center><img src=\"../img/5_deep_learning_computation/macGPUChk2.png\" width=80%></center>\n",
    "    </div>\n",
    "    <div class=\"col col-md-4\">\n",
    "        <center><img src=\"../img/5_deep_learning_computation/macGPUChk3.png\" width=100%></center>\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 选择`PyTorch`版本\n",
    "\n",
    "<center><img src=\"../img/5_deep_learning_computation/pytorchNVIDIA.png\" width=100%></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:36:01.631469Z",
     "iopub.status.busy": "2022-12-07T16:36:01.630909Z",
     "iopub.status.idle": "2022-12-07T16:36:04.378222Z",
     "shell.execute_reply": "2022-12-07T16:36:04.377235Z"
    },
    "origin_pos": 1,
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wed Dec  7 16:36:03 2022       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 460.106.00   Driver Version: 460.106.00   CUDA Version: 11.2     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                               |                      |               MIG M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  Tesla V100-SXM2...  Off  | 00000000:00:1B.0 Off |                    0 |\n",
      "| N/A   27C    P0    47W / 300W |      0MiB / 16160MiB |      0%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   1  Tesla V100-SXM2...  Off  | 00000000:00:1D.0 Off |                    0 |\n",
      "| N/A   34C    P0   244W / 300W |   4321MiB / 16160MiB |     72%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   2  Tesla V100-SXM2...  Off  | 00000000:00:1E.0 Off |                    0 |\n",
      "| N/A   24C    P0    49W / 300W |      0MiB / 16160MiB |      0%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                                  |\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
      "|        ID   ID                                                   Usage      |\n",
      "|=============================================================================|\n",
      "|    1   N/A  N/A     53330      C   ...l-zh-release-0/bin/python     2885MiB |\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "# 使用`nvidia-smi`命令来查看显卡信息\n",
    "\n",
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Mac的M1显卡"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 选择`PyTorch`版本\n",
    "\n",
    "<center><img src=\"../img/5_deep_learning_computation/pytorchMac.png\" width=100%></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### AMD显卡"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 需要开启Windows的Linux子系统\n",
    "- 需要使用`Direct-ML`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "1. 开启Windows的linux子系统(wsl)\n",
    "    - 开启方法参照 [https://learn.microsoft.com/zh-cn/training/modules/wsl-introduction/install-and-setup](https://learn.microsoft.com/zh-cn/training/modules/wsl-introduction/install-and-setup)\n",
    "1. 升级wsl内核\n",
    "    - 找到微软发布的内核更新安装文件 msi [https://www.catalog.update.microsoft.com/Search.aspx?q=wsl](https://www.catalog.update.microsoft.com/Search.aspx?q=wsl)\n",
    "    - 下载最新版本的 wsl_update_x64.msi \n",
    "    - 彻底关闭当前的 WSL，更新内核 —— 以管理员身份运行 `wsl --shutdown`；运行第2步下载的文件，完成内核更新\n",
    "    - 再次开启 WSL，检测内核版本 —— 以管理员身份运行 `wsl， uname -a`\n",
    "1. 在Windows系统中安装AMD显卡驱动程序(针对wsl)\n",
    "    - 显卡驱动地址 [https://www.amd.com/en/support/kb/release-notes/rn-rad-win-wsl-support](https://www.amd.com/en/support/kb/release-notes/rn-rad-win-wsl-support)\n",
    "1. 进入wsl子系统，安装AMD的ROCm\n",
    "    - 安装方法 [https://rocmdocs.amd.com/en/latest/deploy/linux/quick_start.html](https://rocmdocs.amd.com/en/latest/deploy/linux/quick_start.html)\n",
    "1. 进入wsl子系统，安装`miniconda`，并设置一个虚拟环境（`directml`）\n",
    "    - 操作指南[https://learn.microsoft.com/zh-cn/windows/ai/directml/gpu-pytorch-wsl](https://learn.microsoft.com/zh-cn/windows/ai/directml/gpu-pytorch-wsl)\n",
    "    - 上述网页上的Set up a Python environment这一节\n",
    "1. 在虚拟环境`directml`中安装`PyTorch`\n",
    "    <center><img src=\"../img/5_deep_learning_computation/pytorchAMD.png\" width=100%></center>\n",
    "1. 最后装`DirectML`\n",
    "    - `pip install torch-directml`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 计算设备"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 5,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "\n",
    "- 可以指定用于存储和计算的设备，如CPU和GPU\n",
    "- 默认情况下，张量是在内存中创建的，然后使用CPU计算它\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 3,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "- 在`PyTorch`中，每个数组都有一个设备（device），通常将其称为环境（context）\n",
    "- 默认情况下，所有变量和相关的计算都分配给CPU\n",
    "- 也可以指定环境是GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 设置计算环境"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 7,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "- 在`PyTorch`中，CPU环境用`torch.device('cpu')`表示\n",
    "- GPU环境\n",
    "    - 安装了NVIDIA显卡，用`torch.device('cuda')`表示\n",
    "    - mac的M1系统，用`torch.device('mps')`表示\n",
    "    - 安装AMD的显卡，用`torch_directml.device()`表示"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 应该注意的是，CPU设备意味着所有物理CPU和内存，这意味着`PyTorch`的计算将尝试使用所有CPU核心"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- GPU设备只代表一个卡和相应的显存\n",
    "- 如果有多个GPU，我们使用`torch.device(f'cuda:{i}')`来表示第$i$块GPU（$i$从0开始）\n",
    "- 另外，`cuda:0`和`cuda`是等价的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 查看系统中的计算环境"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### NVIDIA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:36:04.382742Z",
     "iopub.status.busy": "2022-12-07T16:36:04.381990Z",
     "iopub.status.idle": "2022-12-07T16:36:07.446633Z",
     "shell.execute_reply": "2022-12-07T16:36:07.445494Z"
    },
    "origin_pos": 10,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(device(type='cpu'), device(type='cuda'), device(type='cuda', index=1))"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "torch.device('cpu'), torch.device('cuda'), torch.device('cuda:1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 13,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 可以查询可用gpu的数量\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:36:07.450762Z",
     "iopub.status.busy": "2022-12-07T16:36:07.450184Z",
     "iopub.status.idle": "2022-12-07T16:36:07.455974Z",
     "shell.execute_reply": "2022-12-07T16:36:07.455020Z"
    },
    "origin_pos": 15,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查询可用gpu的数量\n",
    "\n",
    "torch.cuda.device_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 18,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 定义了两个方便的函数，允许在不存在所需所有GPU的情况下运行代码\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "origin_pos": 20,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(device(type='cpu'), device(type='cpu'), [device(type='cpu')])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def try_gpu(i=0):\n",
    "    \"\"\"如果存在，则返回gpu(i)，否则返回cpu()\"\"\"\n",
    "    if torch.cuda.device_count() >= i + 1:\n",
    "        return torch.device(f'cuda:{i}')\n",
    "    return torch.device('cpu')\n",
    "\n",
    "def try_all_gpus():\n",
    "    \"\"\"返回所有可用的GPU，如果没有GPU，则返回[cpu(),]\"\"\"\n",
    "    devices = [torch.device(f'cuda:{i}')\n",
    "             for i in range(torch.cuda.device_count())]\n",
    "    return devices if devices else [torch.device('cpu')]\n",
    "\n",
    "try_gpu(), try_gpu(10), try_all_gpus()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### AMD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import torch_directml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch_directml.is_available()  # GPU设备是否存在"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch_directml.device_count() # GPU设备数量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Radeon 520\\x00'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "'Intel(R) UHD Graphics 630\\x00'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch_directml.device_name(0)  # GPU设备名称\n",
    "torch_directml.device_name(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- 设置GPU设备"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dml = torch_directml.device(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Apple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='mps')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.device('mps')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 23,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 张量与GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 可以查询张量所在的设备，默认情况下，张量是在CPU上创建的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "origin_pos": 25,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([1, 2, 3])\n",
    "\n",
    "x.device  # 获取该张量的计算环境"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> 需要注意的是，无论何时要对多个项进行操作，它们都必须在**同一个设备上**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 28,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 存储在GPU上"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### NVIDIA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 可以在创建张量时指定存储设备\n",
    "- 例如，在第一个GPU上创建张量`X`\n",
    "- 在GPU上创建的张量只消耗这个GPU的显存。一般来说，需要确保不创建超过GPU显存限制的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "origin_pos": 30,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = torch.ones(2, 3, device=try_gpu())\n",
    "X\n",
    "X.device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 33,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 假设我们至少有两个GPU，下面的代码将在第二个GPU上创建一个随机张量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:36:11.436179Z",
     "iopub.status.busy": "2022-12-07T16:36:11.435624Z",
     "iopub.status.idle": "2022-12-07T16:36:14.699209Z",
     "shell.execute_reply": "2022-12-07T16:36:14.698340Z"
    },
    "origin_pos": 35,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.3821, 0.5270, 0.4919],\n",
       "        [0.9391, 0.0660, 0.6468]], device='cuda:1')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y = torch.rand(2, 3, device=try_gpu(1))\n",
    "Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### AMD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]], device='privateuseone:0')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "device(type='privateuseone', index=0)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X1 = torch.ones(2,3,device=dml)\n",
    "X1\n",
    "X1.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.4452, 0.2311, 0.8754],\n",
       "        [0.8269, 0.8915, 0.6630]], device='privateuseone:1')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 在第二个GPU上创建一个随机张量\n",
    "dml1 = torch_directml.device(1)\n",
    "Y1 = torch.rand(2,3,device=dml1)\n",
    "Y1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Apple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X2_0 = torch.ones(2,3)\n",
    "X2_0\n",
    "X2_0.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]], device='mps:0')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X2 = torch.ones(2,3,device='mps')\n",
    "X2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.6545, 0.2072, 0.0933],\n",
       "        [0.5207, 0.9099, 0.9852]], device='mps:0')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y2 = torch.rand(2,3,device='mps')\n",
    "Y2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 38,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 复制"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 如果要计算`X + Y`，需要决定在哪里执行这个操作\n",
    "- 由于`Y`位于第二个GPU上，可以将`X`传输到第二个GPU并在那里执行操作\n",
    "\n",
    "<center><img src=\"../img/5_deep_learning_computation/copyto.svg\" with=80%></center>\n",
    "\n",
    "- **不要**简单地`X`加上`Y`，因为这会导致异常"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### NVIDIA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:36:14.702836Z",
     "iopub.status.busy": "2022-12-07T16:36:14.702239Z",
     "iopub.status.idle": "2022-12-07T16:36:14.709833Z",
     "shell.execute_reply": "2022-12-07T16:36:14.709036Z"
    },
    "origin_pos": 40,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]], device='cuda:0')\n",
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]], device='cuda:1')\n"
     ]
    }
   ],
   "source": [
    "Z = X.cuda(1)  # 将张量X复制到GPU1\n",
    "print(X)\n",
    "print(Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:36:14.713506Z",
     "iopub.status.busy": "2022-12-07T16:36:14.712939Z",
     "iopub.status.idle": "2022-12-07T16:36:14.719449Z",
     "shell.execute_reply": "2022-12-07T16:36:14.718628Z"
    },
    "origin_pos": 43,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.3821, 1.5270, 1.4919],\n",
       "        [1.9391, 1.0660, 1.6468]], device='cuda:1')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 现在数据在同一个GPU上（`Z`和`Y`都在），可以将它们相加\n",
    "\n",
    "Y + Z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 45,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "- 假设变量`Z`已经存在于第二个GPU上，如果还是调用`Z.cuda(1)`会发生什么？\n",
    "    - 它将返回`Z`，而不会复制并分配新内存"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:36:14.723011Z",
     "iopub.status.busy": "2022-12-07T16:36:14.722460Z",
     "iopub.status.idle": "2022-12-07T16:36:14.727476Z",
     "shell.execute_reply": "2022-12-07T16:36:14.726727Z"
    },
    "origin_pos": 48,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z.cuda(1) is Z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### AMD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "Expected all tensors to be on the same device, but found at least two devices, privateuseone:0 and privateuseone:1!",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[17], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mX1\u001b[49m\u001b[38;5;241;43m+\u001b[39;49m\u001b[43mY1\u001b[49m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: Expected all tensors to be on the same device, but found at least two devices, privateuseone:0 and privateuseone:1!"
     ]
    }
   ],
   "source": [
    "X1+Y1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.8269, 0.8915, 0.6630],\n",
       "        [0.8269, 0.8915, 0.6630]], device='privateuseone:1')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z1 = X1.to(dml1)\n",
    "Z1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.2722, 1.1226, 1.5385],\n",
       "        [1.6539, 1.7831, 1.3260]], device='privateuseone:1')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z1+Y1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z1.to(dml1) is Z1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Apple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "Expected all tensors to be on the same device, but found at least two devices, mps:0 and cpu!",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m/Users/mubojiao/Documents/teaching/machineLearningAI/deepLearning/5_use-gpu.ipynb Cell 72\u001b[0m line \u001b[0;36m1\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/mubojiao/Documents/teaching/machineLearningAI/deepLearning/5_use-gpu.ipynb#Y145sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m X2_0\u001b[39m+\u001b[39;49mY2\n",
      "\u001b[0;31mRuntimeError\u001b[0m: Expected all tensors to be on the same device, but found at least two devices, mps:0 and cpu!"
     ]
    }
   ],
   "source": [
    "X2_0+Y2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]], device='mps:0')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z2 = X2_0.to('mps')\n",
    "Z2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.6545, 1.2072, 1.0933],\n",
       "        [1.5207, 1.9099, 1.9852]], device='mps:0')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Z2+Y2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 50,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 神经网络与GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### NVIDIA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 神经网络模型可以指定设备"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:36:14.730922Z",
     "iopub.status.busy": "2022-12-07T16:36:14.730278Z",
     "iopub.status.idle": "2022-12-07T16:36:14.735655Z",
     "shell.execute_reply": "2022-12-07T16:36:14.734885Z"
    },
    "origin_pos": 52,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "# 下面的代码将模型参数放在GPU上\n",
    "\n",
    "net = nn.Sequential(nn.Linear(3, 1))\n",
    "net = net.to(device=try_gpu())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 55,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 当输入为GPU上的张量时，模型将在同一GPU上计算结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:36:14.738983Z",
     "iopub.status.busy": "2022-12-07T16:36:14.738441Z",
     "iopub.status.idle": "2022-12-07T16:36:15.201841Z",
     "shell.execute_reply": "2022-12-07T16:36:15.201024Z"
    },
    "origin_pos": 56,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.0605],\n",
       "        [-0.0605]], device='cuda:0', grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-07T16:36:15.205621Z",
     "iopub.status.busy": "2022-12-07T16:36:15.204977Z",
     "iopub.status.idle": "2022-12-07T16:36:15.210340Z",
     "shell.execute_reply": "2022-12-07T16:36:15.209602Z"
    },
    "origin_pos": 59,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cuda', index=0)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 确认一下模型参数存储在同一个GPU上\n",
    "\n",
    "net[0].weight.data.device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### AMD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "net1 = nn.Sequential(nn.Linear(3, 1))\n",
    "net1 = net1.to(dml)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.3361],\n",
       "        [-0.3361]], device='privateuseone:0', grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net1(X1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='privateuseone', index=0)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net1[0].weight.data.device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Apple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "net2 = nn.Sequential(nn.Linear(3, 1))\n",
    "net2 = net2.to('mps')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.3047],\n",
       "        [0.3047]], device='mps:0', grad_fn=<LinearBackward0>)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net2(X2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='mps', index=0)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net2[0].weight.device"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "345.994px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
